Image enhancement based on multiple frames and motion estimation

ABSTRACT

A system and method for capturing images is provided. In the system and method, preview images are acquired and global local and local motion are estimated based on at least a portion of the preview images. If the local motion is less than or equal to the global motion, a final image is captured based at least on an exposure time based on the global motion. If the local motion is greater than the global motion, a first image is captured based on at least a first exposure time and at least a second image is captured based on at least one second exposure time less than the first exposure time. After capturing the first and second images, global motion regions are separated from local motion regions in the first and second images, and the final image is reconstructed at least based on the local motion regions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application claiming priority tocopending U.S. patent application Ser. No. 12/397,727 filed on Mar. 4,2009, which patent application claims the benefit of U.S. ProvisionalPatent Application No. 61/035,424 filed on Mar. 11, 2008, now expired,the benefits of which are hereby claimed at least under 35 U.S.C. §120and 35 U.S.C. §119(e); all of the foregoing applications areincorporated herein by reference in their entireties.

FIELD

The invention relates generally to image enhancement and moreparticularly but not exclusively to image enhancement based on multipleframes and motion estimation.

BACKGROUND

Electronic imaging devices image a scene onto a two-dimensional sensorsuch as a charge-coupled-device (CCD), a complementary metal-on-silicon(CMOS) device or other type of light sensor. These imaging devicesgenerally include a large number of photo-detectors (typically two,three, four or more million) arranged across a small two dimensionalsurface that individually generate a signal proportional to theintensity of light or other optical radiation (including infrared andultra-violet regions of the spectrum adjacent the visible lightwavelengths) striking the element. These elements, forming pixels of animage, are typically scanned in a raster pattern to generate a serialstream of data representative of the intensity of radiation striking onesensor element after another as they are scanned. The data acquired bythe image sensor is typically processed to compensate for imperfectionsof the camera and to generally improve the quality of the imageobtainable from the data. Electronic imaging devices generally alsoinclude an automatic exposure control capability that typicallycalculates exposure parameters, such as the exposure time or duration,an aperture size, and gain amount, to result in a luminescence of theimage or succession of images. The exposure parameters typically arecalculated in advance of the picture being taken, and then used tocontrol the camera during acquisition of the image data.

Unfortunately, it is often difficult for the user to hold a camera byhand during an exposure without imparting some degree of shake orjitter, particularly when the camera is very small and light. As aresult, if the exposure time is long, the captured image may have adegree of overall motion blur. Furthermore, even if the camera issteady, a moving object inside the captured scene will be locallyblurred if the exposure time is too long. Accordingly, a common solutionis to adjust exposure parameters based on the overall motion in theimage. However, this can result in at least partially blurred imageswhen multiple sources of motion are present in the scene.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the invention aredescribed with reference to the following drawings, in which:

FIGS. 1A-1C conceptually illustrate a method of forming fused images;

FIG. 2A shows a block diagram of an embodiment of an exemplaryelectronic imaging device operating environment;

FIG. 2B illustrates a block diagram of some of the functional componentsof the controller of FIG. 2A;

FIG. 3 shows a flowchart of exemplary steps in a method for capturingimages;

FIG. 4 shows an exemplary image frame with its pixels grouped intoblocks of multiple pixels used for identifying areas of local motion andmagnitudes of global and local;

FIG. 5 illustrates a flow chart for obtaining a exposure parameter setbased on a motion of interest; and

FIG. 6 shows a flowchart of exemplary steps in a method for fusingimages in accordance with the invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The invention is described with reference to the attached figures,wherein like reference numerals are used throughout the figures todesignate similar or equivalent elements. The figures are not drawn toscale and they are provided merely to illustrate the instant invention.Several aspects of the invention are described below with reference toexample applications for illustration. It should be understood thatnumerous specific details, relationships, and methods are set forth toprovide a full understanding of the invention. One having ordinary skillin the relevant art, however, will readily recognize that the inventioncan be practiced without one or more of the specific details or withother methods. In other instances, well-known structures or operationsare not shown in detail to avoid obscuring the invention. The inventionis not limited by the illustrated ordering of acts or events, as someacts may occur in different orders and/or concurrently with other actsor events. Furthermore, not all illustrated acts or events are requiredto implement a methodology in accordance with the invention.

The term “electronic imaging device”, as used herein, refers to anydevice or portion thereof adapted for at least generating digital imagesof a scene using a light-sensitive sensor (e.g., CCD). Such electronicimaging devices can include, but is not limited to, standalone digitalstill and/or video cameras and digital still and/or video camerasincorporated or connected to devices, systems, and/or communicationsnetworks. For example, electronic imaging devices can be implementedwithin or using computer systems and networks, mobile phones, securitysystems, and vehicles.

In general, to account for motion during capture of an image for ascene, some types of electronic imaging devices acquire one or moreimages of the scene in advance (preview images) and set the exposureparameters accordingly. Accordingly, once the image capture processbegins, an appropriate set of exposure parameters is immediatelyavailable for obtaining the final image. However, even though thisexposure parameter set accounts for current motion in a scene, blurringand/or noise may still be present in one or more portions of the image.

In general, the amount of digital camera image noise will vary accordingto the amplification or gain settings (ISO settings) at the sensorduring image capture. ISO settings are typically used to adjust lightsensitivity at the image sensors during image capture. In low lightconditions or as exposure time is decreased, ISO settings are generallyincreased to compensate for the lower amount of light the sensors willbe exposed to. Unfortunately, as the ISO setting is increased, any noiseassociated with the signals generated by the image sensors is alsoincreased by these higher gain settings associated with the increasedISO setting. As a result, shorter exposure times generally result inincreased noise in the captured image. Blurring is typically the resultof a conventional handheld digital cameras being typically configured toprovide an exposure parameter set that accounts only for one source ofmotion in the scene. For example, if the exposure parameter set providesa relatively long exposure time, then moving objects may appear blurredwhile other objects (e.g., static objects) are relatively clear. As aresult, depending on the exposure parameter set, some blurring of movingobjects and/or noise may be present in the image.

To overcome these limitation, embodiments of the invention provide forobtaining images of a scene that include multiple sources of motion withreduced blurring and noise. In particular, a composite final image isgenerated by combining of a first image captured using a first exposureparameter set accounting for global or overall motion in the scene,which for example can be the result of jitter or hand movement of ahandheld camera, with one or more additional images captured using oneor more additional exposure parameter sets based on moving objects inthe scene. This process is conceptually described with respect to FIGS.1A-1C.

FIG. 1A shows the result of capturing a first image 102 of a sceneincluding buildings 104 and a moving vehicle 106 using a first exposureparameter set. Assuming that the first exposure parameter setcompensates for global motion, such an exposure parameter set wouldspecify a relatively large exposure time. As a result, a moving object,such as vehicle 106, can appear blurred (represented by black fill ofvehicle 106) as compared to static objects, such as buildings 104. FIG.1B shows the result of capturing a second image 108 of the same sceneincluding buildings 104 and moving vehicle 106 using a second exposureparameter set. Assuming that the second exposure parameter setcompensates for motion of vehicle 106 by specifying a relatively shortexposure, the blurring of vehicle 106 in image 108 is reduced relativeto image 102. However, the relatively shorter exposure time can resultin increased noise throughout the image, as described above. Forexample, static objects such as buildings 104, which appeared clearly inimage 102, can appear to include noise in image 108 (represented by theblack fill of buildings 104). Although only buildings 104 in image 108are shown to include noise in FIG. 1B, this is solely for illustrativepurposes. One of ordinary skill in the art will understand that allportions of image 108 will include some amount of noise due to theshorted exposure time.

To provide an image with reduced blurring of moving object and reducedimage noise, the various embodiments of the invention provide acomposite image generated from the fusion of the two or more imagescaptured using different exposure sets, such as the exemplary images inFIGS. 1A and 1B. This is conceptually illustrated in FIG. 1C. FIG. 1Cshows composite image 110 resulting from the fusion of first image 102and second image 108. This is generally accomplished by using firstimage 102, which compensates for global motion, to provide imageportions associated with global motion and using second image 108, whichcompensate for local motion, to provide image portions associated withlocal motion. For example, as shown in FIG. 1A, areas outside portion112 of the first image 102 are associated with global motion aregenerally unblurred. In FIG. 1B, portion 112 is associated with localmotion and areas outside portion 112 can be noisy. Combining theunblurred portions of images 102 and portion 112 from image 108, agenerally unblurred image can be provided with reduced noise as comparedto image 108. For example, as shown in FIG. 1C, a final composite image110 is provided in which vehicle 106 is presented with a reduced amountof blurring and building 104 are presented with a reduced amount ofnoise.

Imaging Device Environment

FIG. 2A shows a schematic diagram of an electronic imaging device 200for obtaining and processing composite still and/or video imagesaccording to an embodiment of the invention. Device 200 may include manymore components than those shown. The components shown, however, aresufficient to disclose an illustrative embodiment for practicing theinvention.

As shown in FIG. 2A, device 200 can include an electronic optical system202, an image processing system 204, and input/output (I/O) system 206.Optical system 202 can include lens 208, as shown in FIG. 2A. However,in other embodiments of the invention, a set of lenses can be used. Inoperation, lens 208 forms image 210 of scene 212 on surface of imagesensor 214. In general, light travels through aperture 216 and shutter218 to reach sensor 214. Focus actuator 220 moves one or more elementsof optical system 202 to focus image 210 on sensor 214. Electricaloutput 222 of sensor 214 carries one or more analog signals resultingfrom scanning individual photo-detectors of surface of sensor 214 ontowhich image 210 is projected. Sensor 214 typically contains a largenumber of individual photo-detectors arranged in a two-dimensional arrayof rows and columns to detect individual pixels of image 210. Signalsproportional to intensity of light striking individual photo-detectorsare obtained in output 222 to generate a frame of video data from whichimage 210 may be reconstructed.

Signals 222 generated by sensor 214 are processed using image processingsystem 204. First, analog signals 222 can be applied to ananalog-to-digital (A/D) converter circuit 224 in image processing system204 that generates digital data signals 226 representing image 210.Digital data signals 226 can then be processed by a processor orcontroller 228 in image processing system. Various functions of imageprocessing system 204 can be implemented using one or more processingelements. These processing elements can be implemented in hardware,software, or any combination thereof. For example, in one embodiment ofthe invention, functions of controller 228 and A/D converter circuit 224can be implemented in one or more integrated circuit chips. Furthermore,in some embodiments of the invention, A/D converter circuit can beincorporated into sensor 214.

Image processing system 204 is communicatively coupled to I/O system 206to allow storage of captured and/or processed image data and to providecontrol signals for electronic imaging device 200. For example, as shownin FIG. 2A, controller 228 is communicatively coupled to a memoryinput/output (I/O) interface 230 and control I/O interface 232 viaconnections 234 and 236, respectively. Control I/O interface 232 can becoupled to user controls or indicators 238 for exchanging controlsignals 240 for electronic imaging device 200. Image processing system204 can also be implemented on one or more computing devices, internalor external to device 200, such as the image processing system describedwith regard to FIG. 2A.

Memory I/O interface 230 can be coupled to a memory element 242 forexchanging data signals 244. Although a single external memory element242 is shown in FIG. 2A, the various embodiments of the invention arenot limited in this regard. In some embodiments of the invention,controller 228 can be coupled to multiple internal and/or externalmemory elements. For example, controller 228 can be coupled or caninclude internal non-volatile memory elements for storing calibrationdata and the like and/or internal volatile memory elements for temporarydata storage. External volatile and/or non-volatile memory elements canalso be coupled to processor 228 for storing image data for transferringimage data to other electronic imaging devices or image processingsystems. Memory element 242 can include, but is not limited to,semiconductor-based memory devices, magnetic storage media devices, oroptical storage media devices.

In addition to being coupled to interfaces 230 and 232, controller 228can also be coupled to control and status lines 246. Lines 246 are, inturn, can be coupled to aperture 216, shutter 218, focus actuator 220,sensor 214, A/D converter 224, and other components of electronicimaging device 200 to provide synchronous operation. Signals in lines246 from processor 228 drive focus actuator 220, set size of opening ofaperture 216, operate shutter 218, and adjust a gain amount for analogsignals 222 at A/D converter 224. A clock circuit 248 can be providedwithin electronic imaging device 200 for providing clock signals 250 tosynchronize operation of the various components. Although shown in FIG.2 as being generated by a separate component in electronic imagingsystem 200, the clock signal for system 200 can be generated withinprocessor 228 or be provided by an external source.

FIG. 2B illustrates a block diagram for particular components that maybe included within controller 228 as previously shown in FIG. 2A. Asshown in FIG. 2B, controller 228 can include a controller processor unit242, which may be general purpose or dedicated to the tasks herein,performing calculations on the image data and controlling operation ofthe camera. Digital data of successive image frames are received vialines 226 by image input unit 264 of controller 228 and then arecommunicated to other components of controller 228 by connection throughmemory management unit 266. Video data of captured image frames can beoutputted by memory management unit 266 to a memory element interfaceand memory element (not shown) through lines 234. Also, the video datacan be outputted to user controls and/or indicators (not shown) throughcontroller interface unit 268 and over lines 236 (to the I/O interface232 of FIG. 2A).

Memory management unit 266 can also be connected to controller processorunit 262. Controller processor unit 262 can also be connected tocontroller interface unit 268 to enable communication of signals alongcontrol status line 246. In some embodiments, controller 228 can alsoinclude a dedicated motion estimation processor unit 270 to perform atleast some of the calculations to estimate the motion of the image fromdata acquired from successive image frames. Although controllerprocessor unit 262 can be configured to perform such calculations, adedicated processor may be provided in one or more embodiments.

FIG. 3 is a flow chart of steps in an exemplary method 300 for capturingimages using an electronic imaging device. The method can begin withstep 302, in which the electronic imaging device is powered on. Themethod can then continue to step 304. In step 304, two or more previewimages of a scene can be captured. Typically such preview images areutilized to detect the amount of motion in a scene and to select thecorrect exposure parameters, as described in further detail below. Forexample, these preview images can comprise the preview images normallydisplayed in a standalone digital camera LCD display. In the variousembodiments of the invention, at least two preview images are acquired,such as 2, 5, or 10. As the number of preview images is increased, theaccuracy of the estimation of the amount of motion in one or moredifferent portions of the scene can be increased. This acquisition canbe performed at any frame rate, such as 30, 40, or 50 frames per secondor more. The estimation of these amounts of motion will be describedbelow in greater detail with respect to FIG. 4.

In some embodiments of the invention, the preview images can have thesame resolution as the final composite image. In other embodiments ofthe invention, the preview images can have a reduced resolution. In thecase of lower resolution preview images, subsequent motion estimationcomputation intensity and motion estimation accuracy can also be reducedas the number of pixels analyzed is also reduced.

Following step 304, the image data for the N number of preview images isused to detect and estimate amounts of motion in a scene in steps 306.In the various embodiments of the invention, any change in motion of thescene image relative to the photo sensor is detected and quantified bylooking at changes in successive preview images. As a result, vectorsand magnitudes of motion, velocity and acceleration can be calculatedfrom data of the N preview images. In step 306, the global motion in theN images (overall motion) can be detected and the magnitude of motionassociated with global or overall motion can be estimated. Additionally,local motion in the N preview images can also be is detected. Theprocess of determining the magnitude of local and global motion indifferent areas of a scene is conceptually described with respect toFIG. 4.

FIG. 4 conceptually shows an exemplary image frame 400 with its pixelsgrouped into blocks of multiple pixels each, such as blocks 402 and 404,used for determining magnitudes of global and local motion and areas oflocal motion in accordance with an embodiment of the invention. In FIG.4, motion of different portions of the scene being captured relative tothe image frame 400 are indicated by arrows 406 and 408. In theexemplary image, global motion in the image frame 400 is represented bya global motion vector M_(G) indicated by arrows 406 in each of theblocks of pixels. The magnitude and direction of M_(G) can be calculatedfrom image data of N preview images by detecting and quantifying overallmovement between the N preview images. Such calculation can beimplemented via a software and/or hardware means. Although vector M_(G)indicates a lateral motion for image frame 400 in FIG. 4, embodiments ofthe invention are not limited in this regard. Rather, global motion canoccur in any direction and in any pattern.

Also as shown in FIG. 4, the image frame 400 includes blocks 404 havingadditional local motion vectors M_(L) independent of the global motion.These vectors represent local motion in the scene being imaged. Suchmotion can be from the movement of a person, a vehicle, and/or otherobjects moving in the scene. Any algorithms for the calculation ofmotion within a scene based on a sequence of images can be used. Forexample, motion vectors can be calculated in a manner that is similar tothe calculation of motion vectors used by known video compressionalgorithms, examples being those of the Moving Picture Experts Group,such as MPEG-4. The vectors M_(G) and M_(L) can therefore be used toestimate both the magnitudes of global and local motion in the scene.

Referring back to FIG. 3, method 300 can detect whether an image capturecommand has been received at step 308 subsequent to or in combinationwith step 306. For example, in the case of a hand held digital camera,the method 300 can determine in step 308 if the shutter button has beenpressed. If no capture command has been received at step 308, the methodcan repeat steps 304 and 306 until the shutter button is pressed.

Once the shutter button has been pressed, the difference between themagnitudes of global and local motion estimated at step 308 can becompared at step 310. If at step 310 the magnitude of the local motionin at least one portion of the scene is not greater than the magnitudeof global motion, then no blurring would typically occur if exposureparameters, including exposure time, are configured based on globalmotion. At step 312, an exposure parameter set, including exposure time,is calculated based on the amount of global motion in the previewimages.

In general, an exposure parameter set is calculated to provide anaverage luminescence across the image within a predefined range. In thevarious embodiments of the invention, any algorithm for calculation ofthe exposure parameter set can be used. As described above, thecalculated exposure parameter set can provide an exposure time orduration and other exposure parameters, such as size of the apertureopening and gain. Generally, an exposure parameter set specifies a largevalue for slow motion, such as global motion, and a smaller value forfaster motion, such as local motion.

The exposure parameter set obtained at step 312 can then be used at step314 to capture a final image. The image data for the final image canthen be stored, as described above with respect to FIG. 2. The method300 can then return to step 304 to prepare for capturing a next image.

If the magnitude of the local motion in at least one part of the sceneat step 310 is greater than the magnitude of the global motion, a fusionor reconstruction process can be used to obtain a composite image, asdescribed above with respect to FIG. 1. First, at step 316, exposureparameter sets can be calculated. In particular, first and secondexposure parameter sets with long and short exposure times,respectively, can be calculated. That is, a first exposure parameter setis calculated for capturing a first image based on the amount of globalmotion, similar to step 312. Subsequently or in combination, a secondexposure parameter set is calculated for capturing a second image basedon the amount of local motion.

Although only two exposure parameter sets are calculated in method 300for capturing two images, in other embodiments of the invention, anynumber of exposure parameter sets can be computed at step 316. Forexample, based on the number of portions of the scene having a magnitudeof local motion greater than the magnitude of global motion in thescene. In such embodiments, a different exposure parameter set can becomputed for each portion of the scene having local motion. In otherembodiments of the invention, exposure parameter sets can be calculatedif at least one portion of the scene has a local motion within anassociated range. In still other embodiments of the invention, to reducethe number of capture images to be combined into a composite image, thesecond exposure parameter set can be based on an average, median, ormaximum amount of motion in the scene.

Once the first and second exposure parameter sets have been calculatedat step 316, respectively, two images can be captured at step 318 usingthe long and short exposure times provided by the first and secondexposure parameter sets, respectively. Once the images are captured atstep 318, the images are processed at step 320 to separate regions ofglobal motion from regions of local motion in the images.

After the separation process is completed at step 320, the final imagecan be reconstructed at step 322. In particular, as described above withrespect to FIG. 1C, areas of the first image associated with globalmotion can be combined with areas of the second image associated withlocal motion. The resulting image therefore combines image informationfrom the first and second images. This can also be referred to as afusion process. Details of an exemplary fusion process will be describedbelow with respect to FIG. 6. The image data for the composite image canthen be stored as the final image. The method 300 can then return toprevious processing at step 304. For example, the method 300 can repeatstarting with step 304 in preparation for capturing a next image.

In general, the exposure parameter sets for global and local motion areobtained by modifying a current or default exposure parameter set in anelectronic imaging device. FIG. 5 is a flow chart showing steps in anexemplary method 500 for calculating an exposure parameter set frommotion quantities according to an embodiment of the invention. Themethod 500 begins at step 530 and continues on to step 532. At step 532,the instant in time to take the picture is estimated from the magnitudeof the motion of interest (global or local motion) previouslycalculated. This estimate is made by extrapolating the magnitude of themotion of interest calculated from the preview images, and thenidentifying the point of zero or minimal motion within a set period. Itis at that instant that the image can be scheduled to be captured. If,however, a zero or minimal motion point cannot be detected with highprecision, due to the complexity of the motion, or if the user haschosen to turn off the delayed capturing option, or if the motionquantities show that there is little or no motion of the image, then thetime for capturing the image is not postponed and capture of the imageis executed right away.

Once the best exposure start time is calculated in step 532, the method500 determines at step 534 whether the current exposure parameter setneeds to be altered based on the exposure start time. For example, ifthe exposure duration in the current exposure parameter is set to bebelow a certain threshold, then no further decrease of the exposure timeis provided. Similarly, if the aperture and gain in the current exposureset are smaller than corresponding thresholds, then it is not necessaryto consider whether motion in the image is small enough to allow theexposure duration to be increased in order to lower them to improvedepth of field or reduce noise. In such a case, the processing proceedsto step 538 and returns to previous processing. That is, the motionquantities are not used or referenced and the picture can be taken rightaway.

However, if the parameters in the current exposure set are not withinoptimum ranges at step 534, they are adjusted at step 536 in order tooptimize them for the magnitude of the motion of interest. For example,as exposure time is decreased over that specified in the currentexposure set (in response to an increased amount of motion); at leastsome amount of gain increase (i.e., an increased ISO setting) isgenerally needed to compensate for the loss in luminescence due to thereduced exposure time. As motion continues to increase, an increase inaperture size can also be needed to compensate for the loss inluminescence. In contrast, as exposure time is increased over thatspecified in the current exposure set (in response to a decreased amountof motion); at least some amount of aperture size reduction is generallyneeded to compensate for the increase in luminescence due to theincreased exposure time. As motion continues to decrease, a decrease ingain can also be needed to compensate for the increase in luminescence.Once the modified exposure parameter set is obtained in step 536, themethod 500 can end at step 538 and return to previous processing.

In the examples of ISO setting adjustment given above, the gain level ofthe analog signal is adjusted before digitizing the signal and thisadjusted level is then used during capture of the image. In addition tothis, or in place of it, the gain of an image can be adjusted in thedigital domain after data of the image have been captured and digitized.For example, the digital gain adjustment can be performed after imagecapture, as part of image processing or enhancement stage, but beforewriting it to a memory element. Although digital gain increase usuallyresults in a noisier image than analog gain increase, it may beconvenient to control the digital gain as well. The amount of digitalgain that is required may be determined during the process that adjuststhe exposure parameters in advance of image capture, as part ofcalculating the exposure time, aperture and perhaps analog gain, butthen applied to the digital data of the image after it has already beencaptured. Alternatively, or in addition, the gain level of the image maybe determined and adjusted as part of the post-processing or enhancementof the captured image data and applied thereafter in that stage.

As previously described, once images have been captured using exposureparameter sets for global and local motion, the images can be fused. Onemethod of fusing image is selecting portions of first and second imagesand combining them into a third image. For example, areas of globalmotion from a first image can be selected and areas of local motion froma second image can be selected. Another method of reconstructing thefinal image is to fuse a portion of an image captured using parametersbased on local motion (local motion image) into an image captured usingexposure parameters based on global motion (global motion). FIG. 6illustrates such a method. In such a method, the global motion image isused as the base image for several reasons. First, since the globalmotion image is captured using an exposure parameter set associated withthe lowest amount of motion in the image, the depth of field is thegreatest. Second, since the gain is typically the lowest for imageshaving lower amounts of motion, such as the global motion image, theamount of noise introduced into the composite image is limited.Accordingly, if a majority of the scene is associated with only globalmotion, the quality of the composite image is only partially affected bythe increased noise and reduced depth of focus in the portions of thelocal motion image therein.

FIG. 6 is a flowchart showing steps in an exemplary method 600 forfusing captured images according to an embodiment of the invention. Themethod 600 can begin at step 640 and continue on to step 642. At step642, a registration or alignment relationship between the capturedimages can be obtained. That is, a function is obtained that describesthe location of pixels in a first captured image relative to a secondcaptured image. Subsequently or in combination with step 642, the localmotion image can be selected at step 644.

Following selection of a local motion image at step 644, one or moreportions of the selected local motion image associated with local motioncan be identified at step 645. That is, with reference to FIG. 4, areasin the local motion image can be identified that correspond to theblocks of local motion (blocks 404). Once the portions of the localmotion image associated with the local motion are identified in step645, the corresponding portion of global motion image can be identifiedat step 646.

After the corresponding portions of the global motion and local motionimages are identified at steps 645 and 646, the identified portions ofthe global motion image are replaced with the corresponding portions ofthe local motion image in step 648. Once the replacement process in step648 is completed, the method 600 can return to previous processing atstep 650.

The replacement process at step 648 can vary depending on theregistration between the images. For example, in some cases, a straightpixel by pixel replacement method can be used. That is, the values ofpixels in the global motion image are directly replaced with values fromcorresponding pixels in the local motion image. Such a method can workwell when the electronic imaging device is attached to a fixed position(e.g., tripod). However, if some amount of global motion is present, astraight pixel by pixel replacement can result in visiblediscontinuities in the composite image, particularly along the edge ofthe inserted portion in the global motion image. This can be a result ofthe position of the pixels not overlapping significantly. That is, theposition of a pixel in the local motion image in the global motion imagelies between the positions of two or more adjacent pixels in the globalmotion image. As a result, the values for pixels being replaced in theglobal motion image can require some amount of correction. For example,the values for the pixels can be adjusted (i.e., by averaging orsmoothing of pixel values) to provide a continuous transition at theedges of the inserted portion. Another option is to calculate sometransformation between the two images according to registration resultsand then warp the local motion image with the resulted transformation. Aprojective transformation is just one of many types of such atransformation.

Additionally, a discontinuity can be visible even if the pixels in thedifferent images are perfectly aligned. Since the images to be combinedare captured using different exposure parameter sets, the brightness andcontrast in the inserted portion can be significantly greater than thatof the global motion image. Therefore, in some embodiments of theinvention, a correction can be applied to the pixels being inserted onthe global motion image to reduce this difference. For example, thevalues of the pixels in the different images can be normalized based onthe exposure time difference. In another example, the values of thepixels in the inserted portion can be normalized based on pixel valuesof the surrounding global motion image.

While various embodiments of the invention have been described above, itshould be understood that they have been presented by way of exampleonly, and not limitation. Numerous changes to the disclosed embodimentscan be made in accordance with the disclosure herein without departingfrom the spirit or scope of the invention. Thus, the breadth and scopeof the invention should not be limited by any of the above describedembodiments. Rather, the scope of the invention should be defined inaccordance with the following claims and their equivalents.

Although the invention has been illustrated and described with respectto one or more implementations, equivalent alterations and modificationswill occur to others skilled in the art upon the reading andunderstanding of this specification and the annexed drawings. Inaddition, while a particular feature of the invention may have beendisclosed with respect to only one of several implementations, suchfeature may be combined with one or more other features of the otherimplementations as may be desired and advantageous for any given orparticular application.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. Furthermore, to the extent that the terms “including”,“includes”, “having”, “has”, “with”, or variants thereof are used ineither the detailed description and/or the claims, such terms areintended to be inclusive in a manner similar to the term “comprising.”

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

The Abstract of the Disclosure is provided to comply with 37 C.F.R.§1.72(b), requiring an abstract that will allow the reader to quicklyascertain the nature of the technical disclosure. It is submitted withthe understanding that it will not be used to interpret or limit thescope or meaning of the following claim.

What is claimed is:
 1. An imaging device, comprising: an optical systemused to receive images; and a processing element communicatively coupledto the optical system and that is programmed to perform actions,including: receiving a plurality of preview images from the opticalsystem; estimating a global motion and a local motion based on at leasta portion of the plurality of preview images; and when the estimatedlocal motion is less than or equal to the estimated global motion,configuring the optical system to provide a final image based at leaston an exposure time based on the estimated global motion; and otherwise:configuring the optical system to provide a first image based on atleast a first exposure parameter set and a plurality of other imagesbased at least on one or more other exposure parameter sets, separatinga global motion region from a local motion region in the first image,separating a plurality of local motion regions from global motionregions in the plurality of other images, and generating a finalcomposite image from a fusion of the separated global motion region fromthe first image with the separated plurality of local motion regions inthe plurality of other images.
 2. The imaging device of claim 1, whereinat least one exposure parameter includes an exposure time, and whereinat least one exposure time within the one or more other exposureparameter sets is less than an exposure time in the first exposureparameter set.
 3. The imaging device of claim 1, wherein configuring theoptical system to provide the first image comprises computing a firstexposure time based on the estimated global motion, the first exposuretime being included within the first exposure parameter set.
 4. Theimaging device of claim 1, wherein at least one exposure parameterwithin the one or more other exposure parameter sets is computed basedon the estimated local motion.
 5. The imaging device of claim 1, whereingenerating a final composite image further comprises replacing imageinformation in the first image for at least one area of location motionwith image information in one or more of the plurality of other imagesfor at least one are of local motion.
 6. The imaging device of claim 1,wherein at least one exposure parameter in the first and one or moreother exposure parameter sets include an exposure time, a size of anaperture opening, or a gain.
 7. A controller device, comprising: amemory management unit configured to receive and manage images; and aprocessor unit that is programmed to receive images and perform actions,including: receiving a plurality of preview images from the memorymanagement unit; estimating a global motion and a local motion based onat least a portion of the plurality of preview images; and when theestimated local motion is less than or equal to the estimated globalmotion, configuring an optical system to provide a final image based atleast on an exposure time based on the estimated global motion; andotherwise: configuring the optical system to provide a first image basedon at least a first exposure parameter set and a plurality of otherimages based at least on one or more other exposure parameter sets,separating a global motion region from a local motion region in thefirst image, separating a plurality of local motion regions from globalmotion regions in the plurality of other images, and generating a finalcomposite image from a fusion of the separated global motion region fromthe first image with the separated plurality of local motion regions inthe plurality of other images.
 8. The controller device of claim 7,wherein at least one exposure parameter within the one or more otherexposure parameter sets is computed based on at least one of an average,a median, or a maximum amount of motion detected within a scene.
 9. Thecontroller device of claim 7, wherein at least one exposure parameterwithin the one or more other exposure parameter sets is computed basedon a determined of a number of portions of a scene having a magnitude ofestimated local motion that is greater than a magnitude of estimatedglobal motion in the scene.
 10. The controller device of claim 7,wherein generating a final composite image further comprises replacingimage information in the first image for at least one area of locationmotion with image information in one or more of the plurality of otherimages for at least one are of local motion.
 11. The controller deviceof claim 7, wherein at least one exposure parameter in first and one ormore other exposure parameter sets include an exposure time, a size ofan aperture opening, or a gain.
 12. The controller device of claim 7,wherein a time to capture the first image is determined based onestimating a magnitude of motion by extrapolating the magnitude ofmotion from the plurality of preview images, and identifying a point ofminimal motion within a set period.
 13. The controller device of claim7, wherein at least one exposure parameter includes an exposure time,and wherein at least one exposure time with the one or more otherexposure parameter sets is less than an exposure time in the firstexposure parameter set.
 14. The controller device of claim 7, whereinconfiguring the optical system to provide the first image comprisescomputing a first exposure time based on the estimated global motion,the first exposure time being included within the first exposureparameter set.
 15. A non-transitory processor-readable storage device,having processor executing instructions that perform actions,comprising: receiving a plurality of preview images from an opticalsystem; estimating a global motion and a local motion based on at leasta portion of the plurality of preview images; and when the estimatedlocal motion is less than or equal to the estimated global motion,configuring the optical system to provide a final image based at leaston an exposure time based on the estimated global motion; and otherwise:configuring the optical system to provide a first image based on atleast a first exposure parameter set and a plurality of other imagesbased at least on one or more other exposure parameter sets, separatinga global motion region from a local motion region in the first image,separating a plurality of local motion regions from global motionregions in the plurality of other images, and generating a final imagefrom a composite of just the separated global motion region from thefirst image with the separated plurality of local motion regions in theplurality of other images.
 16. The storage device of claim 15, whereinat least one exposure parameter within the one or more other exposureparameter sets is computed based on at least one of an average, amedian, or a maximum amount of motion detected within a scene.
 17. Thestorage device of claim 15, wherein at least one exposure parameterwithin the one or more other exposure parameter sets is computed basedon a determined of a number of portions of a scene having a magnitude ofestimated local motion that is greater than a magnitude of estimatedglobal motion in the scene.
 18. The storage device of claim 15, whereingenerating a final image further comprises replacing image informationin the first image for at least one area of location motion with imageinformation in one or more of the plurality of other images for at leastone are of local motion.
 19. The storage device of claim 15, wherein atleast one exposure parameter in first and one or more other exposureparameter sets include an exposure time, a size of an aperture opening,or a gain.
 20. The storage device of claim 15, wherein a time to capturethe first image is determined based on estimating a magnitude of motionby extrapolating the magnitude of motion from the plurality of previewimages, and identifying a point of minimal motion within a set period.